Configure consumer tag names/prefixes #265

sldblog · 2016-11-24T14:20:29Z

Allows us to configure the consumer names (tags) that are visible in RabbitMQ.

Tagging with specific names is useful for us because we have over 100 queues and lots of applications communicating. If we aim to be easily identifiable, we can do these things:

define a routing key convention that refers to the application publishing the message
define a queue name convention that identifies the consumers
identify consumers by tag names

Out of the above three we cannot do the last one. So this changeset enables that configuration :)

How-to

Hutch::Config.set(:consumer_tag_prefix, 'update-dispatcher')

Notes

The default tag name changes from bunny to hutch with this change.

Bonus

Explains on how to (still manually) update the generated configuration section in the readme.

Examples

Default (before the PR)

Default (after the PR)

Custom prefix

michaelklishin · 2016-11-24T14:34:57Z

@olleolleolle thoughts? This looks reasonable to me.

olleolleolle · 2016-11-24T14:41:11Z

README.md

+Generate with
+
+0. `yard doc lib/hutch/config.rb`
+0. Copy the _Configuration_ section from `doc/Hutch/Config.html` here, with the anchor tags stripped.


Thank you!

♥️ 💛 💚 💙 💜

(This is you coming back to improve the README: 🎩 !)

olleolleolle · 2016-11-24T14:46:08Z

The feature is an improvement! So many thumbs: 👍

The failing test is about supporting MarchHare + Bunny: JRuby support.

LoadError: no such file to load -- bunny/channel

In order to get that working, perhaps we need to not use instance_double()? And not only require stuff that's only for 1 kind of Ruby.

olleolleolle

Hm, MarchHare perhaps has an alternate way of expressing this?

olleolleolle · 2016-11-24T14:56:57Z

lib/hutch/worker.rb

+
+    def consumer_tag(queue)
+      prefix = Hutch::Config[:consumer_tag_prefix]
+      queue.channel.generate_consumer_tag(prefix)


So, MarchHare does not have the same interface:

https://github.com/ruby-amqp/march_hare/search?utf8=%E2%9C%93&q=generate_consumer_tag

Ah, it does exist, just another way of expressing it: https://github.com/ruby-amqp/march_hare/pull/93/files

sldblog · 2016-11-24T15:02:33Z

Oh I understand why instance_double wasn't used now. I didn't realise the underlying library is different on JRuby. Let me rework some of this. Thanks for the feedback 👍

sldblog · 2016-11-24T15:33:56Z

spec/hutch/worker_spec.rb

@@ -45,9 +45,17 @@
    end

    it 'sets up a subscription' do
-      expect(queue).to receive(:subscribe).with(manual_ack: true)
+      expect(queue).to receive(:subscribe).with(consumer_tag: %r(^hutch\-.{36}$), manual_ack: true)


Do I need to double check AMQP/RabbitMQ spec for the maximum ~~available~~ length of the consumer tag name?

That would be a cool touch.

(Then we could figure out ways to validate configuration options. Example addition to the Config options' DSL: max_length: 34.)

It "depends", it seems: https://www.rabbitmq.com/amqp-0-9-1-reference.html#basic.consume.consumer-tag it is a shortstr, defined as "shortstr shortstr [short string]"

Seems like it's 255 bytes max: https://github.com/rabbitmq/rabbitmq-common/blob/rabbitmq_v3_6_0/codegen.py#L178-L179 ... I hope that's the same in 3.4.x as well :)

So, now we know that the longest prefix that can be used is: 256 - 1 - 36 = 219 bytes.

Ideally Hutch should loudly complain about prefixes that are longer, e.g. log an error and trim the prefix or fail to start.

Opted for an "on-create" validation as validating the actual prefix length is weird (why 219 bytes?) and coupled to the implementation.

Is the on-create validation sufficient/ok?

sldblog · 2016-11-24T15:43:32Z

lib/hutch/worker.rb

+    def unique_consumer_tag
+      prefix = Hutch::Config[:consumer_tag_prefix]
+      unique_part = SecureRandom.uuid
+      "#{prefix}-#{unique_part}"


I ended up generating this myself. Quote from the commit:

Due to the interface differences between bunny (used for CRuby/MRI)
and march_hare (used for JRuby), it's not possible to consistently use
the generator function queue.channel.generate_consumer_tag for the
unique part of the consumer tags.

Instead we can generate the unique part ourselves without relying on
internals of these gems. Using SecureRandom.uuid seems like a good
idea.

Is this assumption valid? Is it okay to use SecureRandom here?

It is. SecureRandom is available on JRuby and produces values of a known fixed length (36 bytes).

sldblog · 2016-11-24T15:46:20Z

⌛️ I also need to test these with my local RabbitMQ install to ensure it's not breaking on some other integration problems.

Allows us to configure the consumer names (tags) that are visible in RabbitMQ. Tagging with specific names is useful for us because we have over 100 queues and lots of applications communicating. If we aim to be easily identifiable, we can do these things: - define a routing key convention that refers to the application publishing the message - define a queue name convention that identifies the consumers - identify consumers by tag names Out of the above three we cannot do the last one. So this changeset enables that configuration :) Note: this changes the default tag name from `bunny` to `hutch`.

Due to the interface differences between `bunny` (used for CRuby/MRI) and `march_hare` (used for JRuby), it's not possible to consistently use the generator function `queue.channel.generate_consumer_tag` for the unique part of the consumer tags. Instead we can generate the unique part ourselves without relying on internals of these gems. Using `SecureRandom.uuid` seems like a good idea.

[According to the AMQP specification][amqp-spec] the field is a `shortstr`, which is [255 bytes long][codegen]. [amqp-spec]: https://www.rabbitmq.com/amqp-0-9-1-reference.html#basic.consume.consumer-tag [codegen]: https://github.com/rabbitmq/rabbitmq-common/blob/rabbitmq_v3_6_0/codegen.py#L178-L179

sldblog · 2016-11-25T15:32:15Z

I also need to test these with my local RabbitMQ install to ensure it's not breaking on some other integration problems.

💚

Default (now)

Custom prefix

Oh dear

michaelklishin · 2016-11-28T10:53:16Z

Consumer tags do not change and are rarely used in delivery handlers (probably never in Hutch).

…

On 28 Nov 2016, at 13:49, Dávid Lantos ***@***.***> wrote: @sldblog commented on this pull request. In spec/hutch/worker_spec.rb: > @@ -45,9 +45,17 @@ end it 'sets up a subscription' do - expect(queue).to receive(:subscribe).with(manual_ack: true) + expect(queue).to receive(:subscribe).with(consumer_tag: %r(^hutch\-.{36}$), manual_ack: true) Is the on-create validation sufficient/ok? — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

sldblog · 2016-11-28T10:56:39Z

Consumer tags do not change and are rarely used in delivery handlers (probably never in Hutch).

@michaelklishin Does that mean it's okay to validate them when creating the subscriptions?

michaelklishin · 2016-11-28T11:01:37Z

It does to me.

sldblog · 2016-11-28T11:10:40Z

🎉 🙇

sldblog · 2017-01-31T11:31:50Z

@michaelklishin Would it be possible to release this as 0.24? If it's blocked on something, can I help unblocking it?

michaelklishin · 2017-01-31T11:34:46Z

Yes.

michaelklishin · 2017-02-01T20:15:39Z

0.24.0 is out.

sldblog · 2017-02-02T11:04:48Z

❤️

olleolleolle reviewed Nov 24, 2016

View reviewed changes

sldblog commented Nov 24, 2016

View reviewed changes

sldblog added 3 commits November 25, 2016 09:40

michaelklishin approved these changes Nov 28, 2016

View reviewed changes

michaelklishin self-assigned this Nov 28, 2016

michaelklishin merged commit 7550018 into ruby-amqp:master Nov 28, 2016

sldblog deleted the consumer-tag-prefixes branch November 28, 2016 11:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Configure consumer tag names/prefixes #265

Configure consumer tag names/prefixes #265

sldblog commented Nov 24, 2016 •

edited

Loading

michaelklishin commented Nov 24, 2016

olleolleolle Nov 24, 2016

olleolleolle Nov 24, 2016

olleolleolle commented Nov 24, 2016 •

edited

Loading

olleolleolle left a comment

olleolleolle Nov 24, 2016

olleolleolle Nov 24, 2016 •

edited

Loading

sldblog commented Nov 24, 2016 •

edited

Loading

sldblog Nov 24, 2016 •

edited

Loading

olleolleolle Nov 24, 2016

olleolleolle Nov 24, 2016 •

edited

Loading

sldblog Nov 24, 2016

olleolleolle Nov 24, 2016

michaelklishin Nov 24, 2016

sldblog Nov 25, 2016

sldblog Nov 28, 2016

sldblog Nov 24, 2016 •

edited

Loading

michaelklishin Nov 24, 2016

sldblog commented Nov 24, 2016 •

edited

Loading

sldblog commented Nov 25, 2016

michaelklishin commented Nov 28, 2016 via email

sldblog commented Nov 28, 2016

michaelklishin commented Nov 28, 2016

sldblog commented Nov 28, 2016

sldblog commented Jan 31, 2017

michaelklishin commented Jan 31, 2017

michaelklishin commented Feb 1, 2017

sldblog commented Feb 2, 2017

Configure consumer tag names/prefixes #265

Configure consumer tag names/prefixes #265

Conversation

sldblog commented Nov 24, 2016 • edited Loading

How-to

Notes

Bonus

Examples

Default (before the PR)

Default (after the PR)

Custom prefix

michaelklishin commented Nov 24, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

olleolleolle commented Nov 24, 2016 • edited Loading

olleolleolle left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

olleolleolle Nov 24, 2016 • edited Loading

Choose a reason for hiding this comment

sldblog commented Nov 24, 2016 • edited Loading

sldblog Nov 24, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

olleolleolle Nov 24, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sldblog Nov 24, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sldblog commented Nov 24, 2016 • edited Loading

sldblog commented Nov 25, 2016

Default (now)

Custom prefix

Oh dear

michaelklishin commented Nov 28, 2016 via email

sldblog commented Nov 28, 2016

michaelklishin commented Nov 28, 2016

sldblog commented Nov 28, 2016

sldblog commented Jan 31, 2017

michaelklishin commented Jan 31, 2017

michaelklishin commented Feb 1, 2017

sldblog commented Feb 2, 2017

sldblog commented Nov 24, 2016 •

edited

Loading

olleolleolle commented Nov 24, 2016 •

edited

Loading

olleolleolle Nov 24, 2016 •

edited

Loading

sldblog commented Nov 24, 2016 •

edited

Loading

sldblog Nov 24, 2016 •

edited

Loading

olleolleolle Nov 24, 2016 •

edited

Loading

sldblog Nov 24, 2016 •

edited

Loading

sldblog commented Nov 24, 2016 •

edited

Loading